Updated fields #321
Open
Updated fields #321
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jmwilliams89
requested review from
schmikei,
BinaryFissionGames and
jsirianni
|
Should we create a branch that tracks the current way of doing things? There are agents in the wild that will likely be on Stanza 1.x.x for a while, unable to receive plugin updates. Additionally, there are several works in progress that should probably be merged and then rebased into this branch. I will work on getting them reviewed + merged asap. |
|
Should |
There was a problem hiding this comment.
We talked a bit about this earlier, but I wanted to put some comments on this to capture some concerns with the severity mappings.
The plugin migrator uses these mappings (which we might deviate from):
| Stanza (Integral) | Stanza (alias) | OpenTelemetry (alias) |
|---|---|---|
| 0-9 | default | No mapping/Unknown (OTEL has this severity, but only when no mapping is made) |
| 10-19 | Trace | Trace |
| 20-29 | Debug | Debug |
| 30-39 | Info | Info |
| 40-49 | Notice | Warn |
| 50-59 | Warning | Warn2 |
| 60-69 | Error | Error |
| 70-79 | Critical | Error2 |
| 80-89 | Alert | Error3 |
| 90-99 | Emergency | Fatal |
| 100 | Catastrophe | Fatal4 |
Big questions are;
- What are the mappings we're using here?
- Are we going to be dropping any severities? (mapping two
stanzaseverities to the same otel severity) - Are there any implicit severity mappings in
stanzathat aren't being mapped inopentelemetry-log-collection?
added 2 commits
September 10, 2021 11:56
|
I rebased and updated the test cases the best I could. They wont pass until Stanza is released (and we update |
There was a problem hiding this comment.
One thing I noticed was there was a lot of instances of warning: for the severity keys; I only marked one instance in a comment, but there are quite a few. We need those to be warn:, otherwise we'll have problems loading it in the opentelemetry-log-collection stuff, from what I can tell from the code, anyways.
* rebase the stanza-plugins changes * fix haproxy * fix ubiquiti * fix labels rather than attributes on operator field * oracledb attributes * fix haproxy
* Update regex to parse IPv6 (#334) Update default listener log path * Add HAProxy Plugin (#335) * Add haproxy plugin * Add supported platforms and min stanza version * PR Feedback fixes * Rename frontend_name to frontend_name_transport in regex * for all move operations, check if field is nil before moving. "set log type to haproxy and haproxy.error (not .http / .tcp)" * typoe: nill --> nil * typo, log_format: http --> default Co-authored-by: jsirianni <[email protected]> * Allow DBID to be empty & Correct case matching (#331) * Allow DBID to be empty & Correct case matching The DBID field is able to be empty on some versions of Oracle DB The multiline regex was looking for `Audit File`, but logs have `Audit file` * Switch to line end for multiline with double newline pattern * Fix plugin failure when using inline truncate check * Switch back to a regex parse for record splitting Co-authored-by: jsirianni <[email protected]> * Release 0.0.79 (#336) * 0.0.79 changelog * dbid oracle pr * fix release date * move frontend port to resources (#338) * Add more checks to reduce errors (#337) * Add more checks to reduce errors * Add ac_lite_ap_parser change to changelog for ubiquiti * 0.0.80 changelog Co-authored-by: jsirianni <[email protected]> * rebase the stanza-plugins changes * fix haproxy * fix ubiquiti * fix labels rather than attributes on operator field * oracledb attributes * fix haproxy * update regex to handle {} brackets before http request info (#342) * update regex to handle {} brackets before http request info * haproxy http default log format fix * make change backwards compatible * Adjust parsing further based on more detailed oracle db audit logs (#343) * release 0.0.82 Co-authored-by: Dylan Myers <[email protected]> Co-authored-by: EricWHolt <[email protected]> Co-authored-by: jsirianni <[email protected]> Co-authored-by: jsirianni <[email protected]>
* Rebase * remove end to end workflows for now, we will need to test against the otel version of stanza * remove schema and config tests for now, will re-enable when stanza is using otel * Remove schemas for now * remove test for now * Update plugins/haproxy.yaml Co-authored-by: Keith Schmitt <[email protected]> * label --> attribute Co-authored-by: Keith Schmitt <[email protected]>
* Update regex to parse IPv6 (#334) Update default listener log path * Add HAProxy Plugin (#335) * Add haproxy plugin * Add supported platforms and min stanza version * PR Feedback fixes * Rename frontend_name to frontend_name_transport in regex * for all move operations, check if field is nil before moving. "set log type to haproxy and haproxy.error (not .http / .tcp)" * typoe: nill --> nil * typo, log_format: http --> default Co-authored-by: jsirianni <[email protected]> * Allow DBID to be empty & Correct case matching (#331) * Allow DBID to be empty & Correct case matching The DBID field is able to be empty on some versions of Oracle DB The multiline regex was looking for `Audit File`, but logs have `Audit file` * Switch to line end for multiline with double newline pattern * Fix plugin failure when using inline truncate check * Switch back to a regex parse for record splitting Co-authored-by: jsirianni <[email protected]> * Release 0.0.79 (#336) * 0.0.79 changelog * dbid oracle pr * fix release date * move frontend port to resources (#338) * Add more checks to reduce errors (#337) * Add more checks to reduce errors * Add ac_lite_ap_parser change to changelog for ubiquiti * 0.0.80 changelog Co-authored-by: jsirianni <[email protected]> * update regex to handle {} brackets before http request info (#342) * update regex to handle {} brackets before http request info * haproxy http default log format fix * make change backwards compatible * Adjust parsing further based on more detailed oracle db audit logs (#343) * release 0.0.82 * CI Testing: End to End Tests (#345) * end to end nginx testing * fix format * fix format * use sudo to compare against files from container mount * sleep so stanza can parse, kill stanza when done * try cloning log library * use token to clone lob lib * fix repo name * fix expect and output paths * handle both nginx formats * add apache_http workflow * dump container log * dump container log * 10 second sleep * use jq with diff to prevent formatting issues * sudo * cannot use sudo with redirection to diff, so just format with jq before using diff * Switch back to diff, something else is going on.. * pause and cat raw output before comparing * sudo * fix paths * sort before compare * redirect output * sort and cat * use jtool for comparing json files * use jtool for comparing json files * chmod it * haproxy workflow * add oracledb workflow * single test case for oracledb * mount plugins dir * stop and then get stanza logs * sleep 20 seconds instead of 10, sometimes 10 is not enough * fix log dirg * install jtool in its own step * fix mount * split oracle up. start with alert logs * oracle audit log * upgrade jtool and use skip timestamp for haproxy and oracle * upgrade jtoo * upgrade jtool * pause, stop, logs * listener log, oracle * Handle second {} in http log entry if present (#346) Co-authored-by: jsirianni <[email protected]> * Add tcpudp plugin (#341) * Add tcpudp plugin * Add tcpudp schema and tests * Split into two plugins udp and tcp * Add schema files for tests * Update plugins/tcp.yaml * Update plugins/tcp.yaml * Update plugins/udp.yaml * Update plugins/udp.yaml * Update test/configs/tcp/invalid/invalid_listen_port.yaml * Update plugins/udp.yaml Co-authored-by: jsirianni <[email protected]> Co-authored-by: Joseph Sirianni <[email protected]> * tcp / udp: move to message field (#347) * move to message field * \n * Add common event format plugin (#328) * Add common event format plugin * use key value parser for parsing extensions field * Promote fields to labels and resources * Promote fields to labels and resources * Update changelog * Remove key value parser * Add promote device_vendor and device_version to resources Co-authored-by: jsirianni <[email protected]> Co-authored-by: jsirianni <[email protected]> * release 0.0.83 * Add http plugin (#352) * Add http plugin * Update log_type label to http * remove duplicate param * tcp --> http * typo * token_header --> auth_header * small refactor * upgrade stanza 1.2.9 Co-authored-by: jsirianni <[email protected]> * Update Titles and uwsgi field name (#350) Co-authored-by: Joseph Sirianni <[email protected]> * Update cisco_meraki plugin to use key_value_parser (#349) * Update cisco_meraki plugin to use key_value_parser instead of custom regex * use stanza 1.2.9 Co-authored-by: jsirianni <[email protected]> * Create Sonicwall log parser plugin (#340) * Create Sonicwall log parser plugin * Add pri field severity_parser * Rename msg field to message * Add parameter location to support setting timezone * Update to use udp_input and add extra tests * Use stanza 1.2.7 for tests * update stanza and get go.sum * Update plugins/sonicwall.yaml Co-authored-by: Joseph Sirianni <[email protected]> * Fix using wrong parameter if you defined listen_port Co-authored-by: jsirianni <[email protected]> Co-authored-by: Joseph Sirianni <[email protected]> * release-0.0.84 * fix ci link * remove start_at reference * remove start_at test for sonicwall, not needed * enable new operators * fix start_at for w3c tests due to delete_at_end being added * Add cisco_catalyst plugin (#351) * Add cisco_catalyst plugin * Add severity field group to regex. Update parse from field for severity and regex. * Remove parse_to message in udp_input and parse_from message in regex_parser * fix ci link * remove start_at reference * remove start_at test for sonicwall, not needed * enable new operators * fix start_at for w3c tests due to delete_at_end being added Co-authored-by: Joseph Sirianni <[email protected]> * remove tests for now, not compatable with otel branch * remove tests for now, not compatable with otel branch * port cisco catalyst to otel * fix cef * fix haproxy * port http * port sonicwall * fix haproxy * Update plugins/cisco_catalyst.yaml Co-authored-by: Keith Schmitt <[email protected]> * rebase oracledb * try and fix severities that were missed Co-authored-by: Dylan Myers <[email protected]> Co-authored-by: EricWHolt <[email protected]> Co-authored-by: Keith Schmitt <[email protected]> Co-authored-by: schmikei <[email protected]>
* port release 0.0.86 * changelog
Update plugins using tcp_input to use otel tls config
* try apache in ci with otel collector * use docker * remove --rm so we can retain container logs * get initial log output * fix format path * upgade jtool to support otel file output format * dont bother sorting * use jtool 0.0.6 * delay before getting initial logs * fix branch target * haproxy ci * nginx ci * otel oracledb * initial k8s * rename * install minikube * start k8s cluster * install conntrack * try deploying to kubernetes * build image * try without docker env * fix yaml manifest indent * fix hostPath volume syntax * use rollout watch * Set timeout * fix deploy, replace volumes by copying to and from container / image * use rel path for dockerfile * Debug * Debug * Debug * Debug * Debug command and args * setup rbac * its working I think :) * add log files using configmap * fix indent * try k8s version matrix * add more k8s versions and define container runtime with matrix * remove containered and rename default to docker * badges * try containerd minikube * containerd minikube requires crictl * containerd minikube requires crictl, install using github repo instructions * fix containerd log file names * get node status after starting minikube * wait until node ready * fix timeout syntax * merge runtimes using matrix * Detect node name * debug * debug * dont fail all jobs in matrix when one fails * descrbie node on failure * use wait script with trap * oof * fix var name * use bash * remove create output dir step * use docker driver instead of none, configure docker client * update comment * configure docker client when building image * push image to minikube instead of using docker specific eval * split build into build and push steps * add older versions of k8s * rename matrix param to 'format' * minikube does not support k8s 1.13 * try apache with matrix * fix find and replace error * try haproxy and nginx with matrix * typo * oracle matrix * move plugin name to matrix in effort to normalize configs * Add aerospike and make configs generic * remove oiq format * remove plugin from matrix and just reference workflow name directly * add apache combined * badges * add apache common * add cassandra * badges * badges * add codeigniter * badges * Add cef * badges * add couchdb * badges * add docker swarm * swarm badge
…nto updated-fields
* fix cisco meraki circular dependency * parse src and dst into ip and port fields * Start 1.0.0 changelog
Removed unnecessary fallback operator from vmware vsphere
* use enum for tls min version. fix friendly name * version http plugin
Implemented fixes for cisco catalyst and meraki
Filter out agent container logs in k8s
TLS Support for rsyslog, syslogng, and syslog plugins Add missing fields for TCP plugin to fully support TLS
* test k8s 1.23 * remove jtool, it is flawed and not reliable for comparing output to expected output
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
This PR should not be merged until stanza is updated.